For example, a 360-degree panoramic image can be generated from 10 images that are successively captured while a digital camera is rotated approximately 36 degrees at a time. That is, a region equivalent to a 36-degree view is cut out from each of the captured images, and the cut-out images are joined together to generate a 360-degree panoramic image.
Specifically, such a panoramic image is generated as described below.
First, positional relationships between adjacent captured images are determined by analyzing the captured images adjacent to one another. The positional relationships are accumulated to determine the direction in which each of the captured images is taken (hereinafter also referred to as the image capturing direction) in a coordinate system based on the first captured image.
The image capturing direction of each captured image in such a coordinate system based on the first captured image, or the positional relationship with respect to each of the captured images in the coordinate system based on the first captured image, can be normally expressed by a 3-by-3 matrix, and is called a homogeneous transformation matrix (homography).
Where the homogeneous transformation matrix expressing the positional relationship with respect to the sth captured image in the coordinate system based on the first captured image is represented by H1,s, a relationship according to the following expression (1) is established.
                    [                  Mathematical          ⁢                                          ⁢          Formula          ⁢                                          ⁢          1                ]                                                                      [                                                                      X                  1                                                                                                      Y                  1                                                                                    F                                              ]                ∝                              H                          1              ,              s                                ⁡                      [                                                                                X                    s                                                                                                                    Y                    s                                                                                                F                                                      ]                                              (        1        )            
In the expression (1), F represents the focal length of the lens at the time of capturing an image.
The expression (1) means that an object reflected by the pixel at a location (Xs, Ys) in the sth captured image is also reflected by the pixel at a location (X1, Y1) in the first captured image. The homogeneous transformation matrix H1,s where s=1 is the identity matrix. Since a method of determining a homogeneous transformation matrix through an image analysis is a known method, the method is not specifically described herein.
A cylindrical surface shown in FIG. 1 is now assumed to be in the coordinate system based on the first captured image. In FIG. 1, an X1-axis, a Y1-axis, and a Z1-axis that are perpendicular to one another are the axes of the X1-Y1-Z1 coordinate system based on the first captured image, and CL11 represents the cylindrical surface.
The cylindrical surface CL11 is the side surface of a cylinder that has the Y1-axis as the central axis and has a radius F. Here, the point where the cylindrical surface CL11 intersects the Z1-axis is the origin, and a position in a Cx-Cy coordinate system that has a Cy-axis parallel to the Y1-axis and a Cx-axis extending along the cylindrical surface CL11 in the X1-Z1 plane is represented by (Cx, Cy). That is, the length of an arc AR11 in FIG. 1 is represented by Cx, and the length of a straight line SL11 is represented by Cy. More specifically, in the drawing, the downward direction is the positive direction of the Cy-axis.
As can be seen from FIG. 1, in the X1-Y1-Z1 coordinate system, the direction from the origin O of the X1-Y1-Z1 coordinate system toward a point (Cx, Cy) is equal to the (X1, Y1, Z1) direction that satisfies the following expression (2).
                    [                  Mathematical          ⁢                                          ⁢          Formula          ⁢                                          ⁢          2                ]                                                                      [                                                                      X                  1                                                                                                      Y                  1                                                                                                      Z                  1                                                              ]                ∝                  [                                                                      F                  ×                                      sin                    ⁡                                          (                      Cx                      )                                                                                                                          Cy                                                                                      F                  ×                                      cos                    ⁡                                          (                      Cx                      )                                                                                                    ]                                    (        2        )            
The expression (1) and the expression (2) also prove the following. The pixel value of the pixel in a position (Xs, Ys) in the sth (1≦s≦10) captured image is mapped (copied) in the position (Cx, Cy) that satisfies the following expression (3) on the cylindrical surface CL11. As the pixel values of respective positions (Cx, Cy) on the cylindrical surface CL11 are determined in this manner, a 360-degree panoramic image can be formed on the cylindrical surface CL11.
                    [                  Mathematical          ⁢                                          ⁢          Formula          ⁢                                          ⁢          3                ]                                                                      [                                                                      F                  ×                                      sin                    ⁡                                          (                      Cx                      )                                                                                                                          Cy                                                                                      F                  ×                                      cos                    ⁡                                          (                      Cx                      )                                                                                                    ]                ∝                              H                          1              ,              s                                ⁡                      [                                                                                X                    s                                                                                                                    Y                    s                                                                                                F                                                      ]                                              (        3        )            
Referring now to FIGS. 2 through 5, the process of forming a panoramic image by mapping respective captured images on the cylindrical surface CL11 is again described.
FIGS. 2 through 5 show projected images placed in a two-dimensional plane in which the cylindrical surface CL11 is developed, or images obtained by mapping captured images. In the drawings, the horizontal direction is the Cx-axis direction.
In FIG. 2, the length (angle) in the horizontal direction of the two-dimensional plane in which the cylindrical surface CL11 is developed is 0 to 360 degrees, and the angle is equivalent to the Cx-coordinate in the Cx-Cy coordinate system.
In FIG. 2, images P1 through P10 are images obtained by mapping first through tenth captured images on the cylindrical surface CL11 according to the above expression (3) where s is 1 through 10, respectively. For example, the image P1 is the image obtained by mapping the first captured image on the cylindrical surface CL11 through a calculation in which the variable s in the expression (3) is 1.
The shaded regions in the images P1 through P10 are the regions in which the pixels of the first through tenth captured images are mapped, or the regions of projected images. The unshaded regions in the images P1 through P10 are the image in which the pixels of the first through tenth captured images are not mapped, as those regions are outside the image capturing range.
For example, the cylindrical surface CL11 is defined based on the first captured image, and therefore, the center position of the image projected on the image P1 is the position at 0 degree.
As the pixel values of the pixels in the regions of the images projected on the images P1 through P10 have been determined, a 360-degree panoramic image is obtained by combining the image data of those 10 images.
For ease of explanation, an offset is applied to the Cx coordinate system so that the left end of the shaded region in the image P1 in FIG. 2 or the left end of the projected image comes to the position of Cx=0. As a result, the images shown in FIG. 3 are obtained.
In the images P1 through P10 shown in FIG. 3, the shaded regions of the projected images are moved by the amount equivalent to the applied offset.
Here, as a coordinate unit, Cx of one round of the cylindrical surface CL11 is assumed to be 36. Cx is closely related to the pixel pitch in the horizontal direction of the 360-degree panoramic image to be eventually obtained.
As one round of the cylindrical surface CL11 is 36, the horizontal (a Cx-axis direction) length of the eventual 360-degree panoramic image is equivalent to 36 pixels. Normally, the horizontal length of an eventual 360-degree panoramic image is equivalent to several thousands of pixels. However, a number as small as 36 is used herein, for ease of explanation.
In a case where the Cx-coordinate unit is defined as above, the horizontal direction of the two-dimensional plane in which the respective images P1 through P10 shown in FIG. 3 are placed is expressed by using the Cx coordinate system, instead of angle. As a result, the images shown in FIG. 4 are obtained.
In FIG. 4, the shaded regions in the respective images P1 through P10 also represent the regions of projected images.
In FIG. 4, a 360-degree panoramic image can be obtained by partially joining the regions of the projected images in the respective images P1 through P10.
Specifically, a 360-degree panoramic image can be obtained by joining regions TR1 through TR10 of the regions of the projected images in the images P1 through P10 as shown in FIG. 5, for example.
In the example shown in FIG. 5, the region TR1 is the region from 10 to 14 in the Cx-coordinate in the image P1. The region TR2 is the region from 15 to 18 in the Cx-coordinate in the image P2. The region TR3 is the region from 19 to 21 in the Cx-coordinate in the image P3. The region TR4 is the region from 22 to 26 in the Cx-coordinate in the image P4.
The region TR5 is the region from 27 to 29 in the Cx-coordinate in the image P5. The region TR6 is the region from 30 to 31 in the Cx-coordinate in the image P6. The region TR7 is the region from 32 to 34 in the Cx-coordinate in the image P7.
Further, the region TR8 is the region at 35 in the Cx-coordinate and the region from 0 to 3 in the Cx-coordinate in the image P8. The region TR9 is the region from 4 to 5 in the Cx-coordinate in the image P9. The region TR10 is the region from 6 to 9 in the Cx-coordinate in the image P10.
As described above, a 360-degree panoramic image is obtained by joining the regions TR1 through TR10.
As a technique related to panoramic images, there is a suggested technique for preventing division of an image of a moving object at an end of a region that is cut out from a captured image when a panoramic image is generated by joining cut-out regions of respective captured images (see Patent Document 1, for example). By using this technique, image defects such as displaying of a divided part of a moving object in a panoramic image can be prevented. Accordingly, high-quality panoramic images can be obtained.